Skip to content

11L INT6 XSA-all + EMA + VE — ttt_bpb 1.1487#1216

Open
SoHarshh wants to merge 9 commits intoopenai:mainfrom
SoHarshh:submission/12L-banked-VE
Open

11L INT6 XSA-all + EMA + VE — ttt_bpb 1.1487#1216
SoHarshh wants to merge 9 commits intoopenai:mainfrom
SoHarshh:submission/12L-banked-VE

Conversation

@SoHarshh
Copy link
Copy Markdown

@SoHarshh SoHarshh commented Apr 1, 2026

12L Banked + Parallel Muon + Value Embeddings — ttt_bpb 1.1571

Best submittable result: ttt_bpb = 1.1571, 16.47MB (seed 1)

Architecture

  • 12L, 512d, 8H/4KV, LeakyReLU(0.5)² MLP 3×
  • Model banking: qo/kv/mlp_up/mlp_down as 3D tensors [num_layers, M, K]
  • Parallel Muon: async reduce-scatter on banked grads, no DDP
  • Value Embeddings: ve_dim=128, last 2 layers
  • EMA(0.997) with QAT-reset at quantization activation
  • INT4 MLP + INT4 bigram + INT6 attn + zstd
  • XSA last 4 layers, Partial RoPE (16/64), LN Scale 1/√(layer+1)
  • Legal TTT, lr=0.002, 3 epochs

Also included: 11L INT6 XSA-all experiment (quality record, unsubmittable)

  • ttt_bpb = 1.1487 — new quality best
  • 19.03MB — over 16MB budget (INT6 all layers less compressible than INT4 under LZMA)
  • GPTQ in progress to bring size below 16MB

SoHarshh and others added 9 commits March 28, 2026 08:04
train_gpt_v2.py:
- LZMA compression support (COMPRESS=lzma env var)
- Full Hessian GPTQ: gptq_quantize_weight() + collect_gptq_hessians()
  (GPTQ_ENABLED=1 activates post-training column-wise quantization)

train_gpt_v3.py (Parallel Muon):
- All replicated-param all_reduces now launched async simultaneously
  so NCCL can pipeline them (saves ~3-5ms/step vs serial blocking calls)
- Removed redundant .contiguous() in non-XSA attention path

run.sh:
- v11_proxy: 1-GPU smoke test for 11L INT6 stack
- v11_int6_xsaall: 11L INT6 + XSA-all + LZMA + VE (train_gpt_v2.py)
- v11_gptq: same + GPTQ_ENABLED=1 (train_gpt_v2.py)
- v11_banked: 11L INT6 + XSA-all + LZMA + VE + Parallel Muon (train_gpt_v3.py)
@MatoTeziTanka
Copy link
Copy Markdown

Community Review — 11L INT6 XSA-all + EMA + VE — ttt_bpb 1.1487

BPB: 1.1487 | Compliance: LOOKS CLEAN — score-first-per-chunk TTT (legal #1416/#1423 pattern)

What I found in the code (head SHA 3da121027713, file records/track_10min_16mb/2026-03-28_12L_INT4_bQAT_VE/train_gpt.py):

The TTT path at line 1017 implements the score-first-per-chunk pattern: each chunk is scored under torch.no_grad() / inference_mode() before the base_model.train() + SGD adaptation runs on that same chunk, with an is_last_chunk guard so the final chunk gets no adaptation pass. This is the structural shape the legal frontier uses (PRs #1416 erichroepke, #1423 aryanbhosale).

Per Issue #402 and Issue #677, TTT is legal when each token is scored before the adapter updates on it, and that's what the code does here — chunk ci is scored under weights adapted only on chunks 0..ci-1. No prequant_ttt_adapt_adamw(val_tokens, ...) multi-epoch fine-tune, no scored-region SLOT, no target-in-key n-gram cache.

CPU smoke test (CT2038 proteus-engine, 2026-04-11): import OK in 0.04s, dim=512, layers=10, vocab=1024, code=78732 B, SMOKE_TEST_PASS

Verdict: LOOKS CLEAN.

Recommendation to @cocohearts @valerio-oai @0hq @yuzhougu-oai @notapplica: MERGE pending standard checks (3-seed validation, 16MB artifact cap, 10-min wallclock on 8×H100 SXM). The compliance picture matches the legal reference frontier and no flags were raised by the classification pass.

Auto-classification caveat: this review was drafted by the AST-based classifier against a template derived from manually-reviewed cluster PRs (#1420, #1450, #1487, #1541, #1529, #1533, #1518). If I've misread a subtlety in your eval path — e.g., multi-epoch TTT that I mistook for single-pass, or a target-in-key lookup I missed in a helper function — please flag it and I'll re-run the audit manually.


Reviewed by @MatoTeziTankaThe Agora. CPU smoke test (CT2038 proteus-engine, 2026-04-11): import OK in 0.04s, dim=512, layers=10, vocab=1024, code=78732 B, SMOKE_TEST_PASS. Classification via deterministic AST-based classify_prs.py (pattern bank derived from ~65 manually-reviewed PRs earlier in the 2026-04-11 sweep). This review was auto-drafted from a template and spot-checked before posting — if the template misread your code, please call it out so I can iterate the classifier.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants